Empirical exposure normalization

ABSTRACT

An automated process empirically normalizes a “dark” image by adjusting the apparent exposure to compensate for nonlinearity in the luminance response of the image sensor. The process includes receiving at least two digital images, one of the digital images having an exposure value that is greater than that of another of the digital images. A reduced-resolution pair of images is produced from the at least two digital images. At least one representative scale factor is calculated from tonal values in the two images and at least one empirical scale factor is determined by selective interpolation between the representative scale factor and a comparative scale factor. The empirical scale factor is used in a function applied pixelwise to the darker of the digital images to produce an empirically normalized digital image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/791,500, filed on Oct. 24, 2017, now allowed, the contents of whichare incorporated herein by reference.

TECHNICAL FIELD

This disclosure relates generally to the field of digital imageprocessing. More specifically, this disclosure relates to processingtechniques to normalizing the exposure of a digital image.

BACKGROUND

The ready availability of camera technology, particularly in small,lightweight computing devices, has increased the prevalence of amateurphotography. Hardware and imaging software in both computing devices anddedicated digital cameras have advanced so that photographers of alllevels have access to highly advanced image processing that can beapplied to images in many ways and at any time, even immediately afterimage capture on a device. One area in which image processing stillsometimes falls short is when image processing is applied to imagestaken at or near the limits of the dynamic range of the imaging sensorbeing used. Such images might be taken, as an example, during exposurebracketing for high dynamic range (“HDR”) photography or for imageselection.

In order to adjust the exposure of such images during furtherprocessing, existing applications typically refer to exchangeable imagefile (“EXIF”) data within the image files. EXIF data values aretypically assigned based on camera settings assuming that the luminanceresponse of a digital image sensor is linear, or nearly so. In practicehowever, most digital image sensors become nonlinear near their limits.For example, most digital image sensors are nonlinear in their responseto light at levels that are very low relative to the sensor's nativesensitivity. Thus, it is difficult to achieve optimum results when usingthe “dark” image of a series of images captured of a scene for imageenhancement, for example, to form a combined image through HDRphotography.

SUMMARY

A method of empirically normalizing the exposure value of a digitalimage includes receiving at least two digital images, one of the digitalimages having an exposure value that is greater than another of thedigital images, the at least two digital images including at least someidentical photographic content. A reduced-resolution pair of images isproduced from the at least two digital images. At least a light tonerepresentative scale factor is calculated from the reduced-resolutionpair of images, and at least a light-tone empirical scale factor iscalculated by selectively interpolating between the light-tonerepresentative scale factor and a normal scale factor. An empiricallynormalized digital image is generated by applying, pixelwise, a functionincluding the light-tone empirical scale factor to at least one of theat least two digital images.

These illustrative embodiments are mentioned not to limit or define thedisclosure, but to provide examples to aid understanding thereof.Additional embodiments are discussed in the Detailed Description, andfurther description is provided there.

BRIEF DESCRIPTION OF THE DRAWINGS

Features, embodiments, and advantages of the present disclosure arebetter understood when the following Detailed Description is read withreference to the accompanying drawings, where:

FIG. 1. is a schematic process diagram depicting an example of twodigital images being processed to achieve empirical exposurenormalization of the darker of the two images, according to certainembodiments.

FIG. 2 is a block diagram schematically illustrating components of anexample electronic device that is capable of empirical exposurenormalization, according to certain embodiments.

FIG. 3 is a diagram depicting an example of an environment in which anexposure normalization system is practiced, according to certainembodiments.

FIG. 4 is a flowchart depicting an example process for empiricalexposure normalization, according to certain embodiments.

FIG. 5 is a flowchart depicting an additional example process forempirical exposure normalization, according to certain embodiments.

FIG. 6. is a graph illustrating various mathematical functions that canbe used for empirical exposure normalization, according to certainembodiments.

DETAILED DESCRIPTION

When multiple digital photographs with differing exposure values arecaptured of a given scene, an image or images at or near the limits ofthe dynamic range of the imaging sensor may be exposed differently thancamera settings would suggest. Such images might be taken, as anexample, during exposure bracketing for high dynamic range (HDR)photography or for image selection. In particular, often a darker imageis not exposed as expected because the darker image is captured suchthat the ambient light is in a luminance range for the image sensorwhere its luminance sensitivity is nonlinear.

Techniques are disclosed herein can address this issue by empiricallynormalizing the exposure of a digital image or digital images, therebyproviding accurate results when working with multiple images of the samescene wherein each image has a different exposure value (“ev”). Suchimages can be created, for example, by changing at least one of theparameters in the “exposure triangle” of parameters, namely, sensitivity(“ISO”), focal ratio (“f-stop” or loosely “aperture”) and time (“shutterspeed”) with each image capture. More particularly, the techniquesdiscussed in the present disclosure provide an automated process foradjusting the apparent exposure of a “dark” image of a series of digitalimages to compensate for nonlinearity in the luminance response of theimage sensor. Such an adjustment can make the dark image's exposurestatistically fit more closely within a desired linear relationship ofbrightness across images as compared to relying on the dark image(s) ascaptured.

To appreciate a practical example of the automated process describedherein, assume a camera, computer, or other electronic device receivestwo digital images taken with camera settings set to achieve relativeexposure values for each digital image such that one image is darkerthan the other. These two digital images include at least some identicalphotographic content, for example, the two images are two images takenat about the same time of the same scene, the only major differencebeing exposure value. Such images might be captured during imagebracketing or as part of an HDR process. The darker of the two digitalimages is brighter than expected based on the camera settings, andbrighter than desired, because the camera sensor's luminance response isnon-linear at low light levels. To correct the exposure of the darkimage, a reduced-resolution pair of images is produced from the twodigital images. Each of the images in the reduced-resolution pair is acopy of one of the original images, except that it has lower resolution.The processing device performing the various operations of the processcalculates empirical scale factors using light and dark tone samplesfrom both of the reduced-resolution images. Optionally, a camera sensorparameter describing at what light levels the camera sensor becomesnon-linear in its luminance response can be used along with the tonalinformation from the reduced-resolution images to determine theempirical scale factors. A mathematical function of the empirical scalefactors is applied to the darker of the two original digital images toproduce an empirically normalized darker digital image, one with anapparent exposure value corresponding to the camera settings used tocapture the original dark image, thus automatically cancelling out orcompensating for the nonlinearity in the camera sensor luminanceresponse.

Referring now to the drawings, FIG. 1 schematically illustrates theprocess described herein at a high level. Original dark digital image102 is filtered (reduced in resolution) to produce a reduced-resolutiondark image 104 of a reduced-resolution pair of images. Original lightdigital image 106 is filtered (reduced in resolution) to producereduced-resolution light image 108 of the reduced-resolution pair ofimages. Reduced-resolution digital images 104 and 108 are substantiallylower in resolution than original digital images 102 and 106, as can beappreciated by observing the blockier appearance of thereduced-resolution pair of digital images 104 and 108. Pairwise sets ofpixels, such as pixel 110 and pixel 112 (size exaggerated for clarity),are sampled from throughout the reduced-resolution pair of images toproduce a set of pairs of light-tonal values and a set of pairs ofdark-tonal values from both digital images. These sets of tonal valuesare statistically processed to produce a “normalized” version 114 of thedark image. By “pairwise sets” of pixels, what is meant is that eachpair of pixels in a set includes one pixel from each of the pair ofreduced-resolution images, and each pixel of the pair represents thesame visual point in the images. The image pair can be aligned prior tonormalization, for example, by using common features.

“Exposure normalization” (or simply “normalization” in this context) ofthe darker of two images is understood to be applying a mapping to thepixels in the darker image such that the result is perceived to be, onaverage (discounting noise in the darker image and oversaturation in thebrighter image), as bright as the brighter image. This mapping may takethe non-linearity of the sensor response for very low light levels intoaccount. If this mapping takes the non-linearity into account, it couldalso be used to “linearize” (relative to the light image) the sensorresponse at low light levels in the darker image by applying thismapping (after which the relative EV between the two images will beclose to 0) and then uniformly dividing all pixels in the result by thelight-tone empirical scale factor (after which the relative EV betweenthe two images will be close to log₂ (light-tone scale factor)). Thislinearization process will typically result in no noticeable change inthe light tones and a slight darkening of the shadows, since thelight-tone empirical scale factor is typically greater than thedark-tone empirical scale factor. After this linearization process,recomputing the relative response between the linearized dark image andthe light image should now produce a mapping that if plotted would lookcloser to a straight line, as will be discussed later with respect toFIG. 6.

The following examples are provided to introduce the details of thepresent disclosure. In some examples, the described processes can beimplemented by means of computer program instructions (or “code”)executing on a processing device to cause the processing device toperform the methods described.

In one example, a processing device receives at least two digitalimages, one of the at least two digital images having an exposure valuethat is greater than another of the at least two digital images. The atleast two digital images include at least some identical photographiccontent. A reduced-resolution pair of images is produced from the atleast two digital images. A light-tone representative scale factor iscalculated from the reduced-resolution pair of images, and a light-toneempirical scale factor is calculated by selectively interpolatingbetween the light-tone representative scale factor and a normal scalefactor. An empirically normalized digital image is generated byapplying, pixelwise, a function including at least the light-toneempirical scale factor to at least one of the at least two digitalimages.

In one example, the function mentioned above includes a camera sensorparameter. In at least one example, the camera sensor parameter includesa luminance threshold for dark tones. If the luminance threshold for aparticular camera sensor is unknown, it can be determined by iterativelycorrecting images from an initial guess until the resulting exposurenormalization is optimal.

In one example, a dark-tone empirical scale factor is produced bycalculating a dark-tone representative scale factor from thereduced-resolution pair of images and selectively interpolating betweenthe light-tone representative scale factor and the dark-tonerepresentative scale factor to produce the dark-tone empirical scalefactor. The function that is applied to produce the empiricallynormalized digital image then includes both the light-tone empiricalscale factor and the dark-tone empirical scale factor.

In one example, the calculating of the light-tone representative scalefactor includes gathering a light-tone pairwise set of scale factorsfrom the reduced digital images, and the calculating of the dark-tonerepresentative scale factor includes gathering a dark-tone pairwise setof scale factors from the reduced-resolution pair of images.

As used herein, the term “camera” refers to an imaging device that iscapable of capturing a photographic or video image. Unless otherwiseindicated, a camera includes both specialized devices (e.g., deviceswith no general functions other than taking pictures) and multipurposedevices (e.g., devices, such as smart phones, that are capable ofperforming functions besides taking pictures). Examples of camerasinclude, but are not limited to, still-image cameras, video cameras,smartphones, tablet computing devices, webcams, security cameras, andother devices capable of capturing still images or motion footage.

As used herein, the terms “taking a picture,” “capturing,” or comparablephrases, refer to the act of capturing one or more digital images usinga camera. In some embodiments, taking a picture occurs in response to asingle user action. In some cases, multiple images are captured by asame camera at a same time, or substantially the same time, and thecaptured picture is based on more than one of the multiple images. Themultiple images have exposure value, settings, or other differences inthe camera's actions. The embodiments described herein can be readilyapplied to situations in which multiple images of different exposurevalues are being used.

As used herein, the terms “picture” and “photograph” refer to a mediaitem including visual content captured with a camera by a photographer.Unless otherwise indicated, a picture can be based on multiple digitalimages.

As used herein, the term “digital image” means an electronicrepresentation of photographic subject matter, such as by set of pixels.A digital image is of any suitable format, such as a bitmap or a JPEGfile format. For convenience, digital images are referred to herein ashaving settings (e.g., a digital image with a high ISO setting). Unlessotherwise indicated, this refers to digital images taken by camerashaving the setting (e.g., a digital image taken by a camera having ahigh ISO setting). In some cases, digital images are received from acamera in response to a single user action, such as the action of takinga picture. Additionally or alternatively, digital images are receivedfrom a series of related images, such as video data.

As used herein, the terms “photographic content” and “content” refer tothe subject matter of a picture or a digital image, unless otherwiseindicated. It is to be understood that similar subject matter may havevarious appearances (or non-appearance) in various digital images. Asused herein, the terms “ISO,” “ISO setting,” “sensitivity setting,” and“light sensitivity setting” refer to a camera's sensitivity to availableambient light in the environment of the subject matter. A higher ISO (orhigher sensitivity) indicates that a camera is more sensitive to theambient light. Additionally or alternatively, a lower ISO (or lowersensitivity) indicates that the camera is less sensitive to the ambientlight.

FIG. 2 is a block diagram schematically illustrating selected componentsof an example electronic device 200 that includes digital camerafunctionality. Electronic device 200 may include, for example, one ormore devices selected from a smartphone, a tablet computer, a laptopcomputer, a digital camera, or any other computing device capable ofdigitally recording an observed scene. A combination of differentdevices may be used in certain embodiments. In general, the variousembodiments disclosed herein can be implemented in conjunction with awide range of existing or subsequently developed hardware capable ofcapturing and displaying digital images. In FIG. 2, electronic device200 includes, among other things, a processor 210, flash memory 216 anoperating system 220, which at least partially resides in flash memory216, a communication module or modules 230, a random access memory 240,and digital imaging sensor 250. As can be further seen, a bus and/orinterconnect 214 is also provided to allow for inter- and intra-devicecommunications using, for example, communication module 230.

Depending on the particular type of device used for implementation,electronic device 200 is optionally coupled to or otherwise implementedin conjunction with an input/output devices 260 such as one or more of atouch sensitive display, a speaker, a printer, an antenna for wirelessdata communication, a microphone, tactile buttons, and tactile switches.For example, in a particular alternative embodiment wherein electronicdevice 200 is implemented in the form of a tablet computer, certainfunctionality is provided in conjunction with a touch sensitive surfacethat forms part of the tablet computer. Electronic device 200 canoptionally be coupled to a network to allow for communications withother computing devices or resources, such as networked image processingservices and a networked image repository. However such networkconnection is optional, and therefore in certain embodiments, electronicdevice 200 can be understood as being capable of autonomously carryingout exposure normalization on images captured by digital imaging sensor250. Other components and functionality not reflected in the schematicblock diagram of FIG. 2 will be apparent in light of this disclosure,and thus it will be appreciated that other embodiments are not limitedto any particular hardware configuration.

Processor 210 can be any suitable processor, and may include one or morecoprocessors or controllers, such as an audio processor or a graphicsprocessing unit, to assist in control and processing operationsassociated with electronic device 200. Operating system 220 may compriseany suitable operating system. As will be appreciated in light of thisdisclosure, the techniques provided herein can be implemented withoutregard to the particular operating system provided in conjunction withelectronic device 200, and therefore may also be implemented using anysuitable existing or subsequently developed platform. Communicationmodule 230 can be any appropriate network chip or chipset which allowsfor wired and wireless connection to other components of electronicdevice 200 and to a network, thereby enabling device 200 to communicatewith other local and remote computer systems, servers, and resources. Inone implementation, random access memory 240 is used to temporarilystore image data 242 that is being processed, while flash memory 216 isused to store image data 280 long term. Flash memory is also used tostore normalization module 290, which causes processor 210 toempirically normalize an exposure of a digital image. Both image data242 and image data 280 may include input images and the normalized imagethat is output by processor 210 executing normalization module 290.

The embodiments described herein can be implemented in various forms ofhardware, software, firmware, or special purpose processors. Forexample, in one embodiment a non-transitory computer readable medium hasinstructions encoded thereon that, when executed by one or moreprocessors, cause one or more of the exposure normalizationmethodologies described herein to be implemented. The computer readablemedium can be integrated into a digital camera or an electronic deviceincluding a digital camera, such as a smartphone, as in flash memory 216of device 200 of FIG. 2. The instructions can be encoded using anysuitable programming language.

The functionalities disclosed herein can optionally be incorporated intoa variety of different software applications, including mobileapplications installed on a smartphone, tablet computer, compact digitalcamera, digital single lens reflex camera, video camera, or otherportable electronic device. The functionalities described herein canadditionally or alternatively leverage services provided by, or beintegrated into, other software applications, such as digital image ordigital video editing software applications. The computer softwareapplications disclosed herein may include a number of different modules,sub-modules, or other components of distinct functionality, and canprovide information to, or receive information from, still othercomponents and services. These modules can used, for example, tocommunicate with input and/or output devices such as a display screen, atouch sensitive surface, a printer, and any other suitable input/outputdevice. Other components and functionalities not reflected in theillustrations will be apparent in light of this disclosure, and it willbe appreciated that the present disclosure is not intended to be limitedto any particular hardware or software configuration. Thus in otherembodiments the components illustrated in FIG. 2 may include additional,fewer, or alternative subcomponents.

FIG. 3 is a diagram of an exemplary environment 300 in which one or moreembodiments of the present disclosure are practiced. The environment 300includes an image normalization system 310, configured to producenormalized digital images, and a camera, such as camera 320. In someembodiments, the environment 300 also includes one or more of network390, a digital storage device, such as storage device 330, and acomputing device, such as computing device 340. In certain embodiments,one or more of these elements are included by another of the elements.For example, the camera 320 includes the normalization system 310.Additionally or alternatively, computing device 340 includes one or moreof normalization system 310, storage device 330, or camera 320.

In an embodiment, camera 320 provides a set of images 301 to anormalization system 310. In some cases, image set 301 includes digitalimages captured with different exposure values. The image set 301 isprovided via one or more of (without limitation) an internal deviceconnection, network 390, a portable storage device (e.g., a memory key,a digital media card), or using any suitable technique. In some cases,various individual images within set 301 are provided using variousdifferent techniques. The normalization system 310 performs one or moretechniques as described herein to provide empirical normalization of adigital image, and provides an empirically normalized digital image orempirically normalized digital image 304.

The normalization system 310 can produce the empirically normalizeddigital image 304 (and may provide other images instead of or inaddition to the empirically normalized digital image, such as an HDRphotograph made using the empirically normalized digital image) to oneor more receiving devices. The empirically normalized digital image 304is provided via one or more of (without limitation) an internal deviceconnection, network 390, a portable storage device (e.g., a memory key,a digital media card), or using any suitable technique. In someembodiments, the receiving device is camera 320. Additionally oralternatively, the receiving device is one or more of camera 320,computing device 340, or storage device 330. In some cases, theempirically normalized digital image 304 is provided to variousreceiving devices using various techniques. For example, camera 320receives empirically normalized digital image 304 via an internal deviceconnection, and storage device 330 receives empirically normalizeddigital image 304 via network 390.

Any suitable computing system or group of computing systems can be usedfor performing the operations described herein. For example, a processorexecutes computer-executable program code of the normalization system ormodule. Examples of processor include a microprocessor, anapplication-specific integrated circuit (“ASIC”), a field-programmablegate array (“FPGA”), or other suitable processing device. The processorincludes any number of processing devices, including one. A memorydevice for storing the computer program instructions and digital imagesincludes any suitable non-transitory computer-readable. Thecomputer-readable medium includes any electronic, optical, magnetic, orother storage device capable of providing a processor withcomputer-readable instructions or other program code. Non-limitingexamples of a computer-readable medium include a magnetic disk, a memorychip, a ROM, a RAM, an ASIC, optical storage, magnetic tape or othermagnetic storage, or any other non-transitory medium from which aprocessing device reads instructions. The instructions may includeprocessor-specific instructions generated by a compiler or aninterpreter from code written in any suitable computer-programminglanguage, including, for example, C, C++, C #, Visual Basic, Java,Python, Perl, JavaScript, and ActionScript.

FIG. 4 is a flowchart depicting an example of a process 400 forempirically normalizing the exposure of an image, with a computingdevice carrying out the process by executing suitable program code. Atblock 402 of FIG. 4, two digital images are received. One of the twodigital images has an exposure value that is greater than the other. Thetwo digital images include at least some identical photographic content.At block 404, a reduced-resolution pair of images is produced from thetwo digital images. At block 406, the processing device calculates alight tone representative scale factor from the reduced-resolution pairof images. At block 408, a light-tone empirical scale factor is producedby selectively interpolating between the light-tone representative scalefactor and a normal scale factor. At block 410, the processing devicegenerates an empirically normalized digital image by applying,pixelwise, a function including at least the light-tone empirical scalefactor to the darker image.

The input images must be in the same color space, and this color spacemust be linearly related to the sensor. That is, for any pixel, scalingthe incident light at the sensor site(s) from which its color iscalculated results in that pixel being scaled by the same amount,assuming no deviations due to noise or non-linearities in the sensor atvery low or very high light levels. For example, in a JPEG image createdby a camera the pixels would typically be the result of applyingnon-linear processing to the demosaiced sensor values, such as whitebalancing, brightness/contrast adjustment and gamma encoding,collectively known as the camera response function. In order for theJPEG pixels to be linearly related to the sensor, such that exposurenormalization as described here can be done, the inverse camera responsefunction would need to be applied.

FIG. 5 is a flowchart depicting a detailed example of a process 500 forempirically normalizing the exposure of an image, with a computingdevice carrying out the process by executing suitable program code. Forillustrative purposes, the process 500 is described with reference toexamples depicted here. Other implementations are possible. At block502, one of two digital images, a dark exposure d with dimensions w×h,is acquired. The digital images that are input to the normalizationprocess or system are acquired by an image sensor in a computing devicesuch as electronic device 200 of FIG. 2, or by a dedicated camera suchas camera 320 of FIG. 3.

At block 504 of FIG. 5, another of the digital images, a light image 1,with the same dimensions w×h, is acquired. The light image has an evthat is greater than the ev of the dark image. In this example, if thedark image has an ev of v, the light image has an ev of v+δ. Thus anormal scale factor e that might be used in a typical HDR process wouldproduce l that is e brighter than d, where e=2^(δ). Such a scale factoris normally determined with reference to exposure information in EXIFdata. The two digital images include at least some identicalphotographic content. Typically, such images are captured one afteranother (possibly with additional images) while photographing a scenethat is mostly stationary, such as with a typical HDR technique. In sucha case, the photographic content could be set to be substantiallyidentical. By substantially identical, what is meant is that the imagesappear identical other than ev, minor alignment differences that can beeffectively corrected with typical warping techniques, or the presenceof a moving subject that can be removed or made to look clear andfocused with typical de-ghosting techniques.

At blocks 506 and 508, the two digital images are filtered to produce apair of reduced-resolution images. In this example, a new image of sizew′×h′ is produced for each of the original digital images. The lighterimage of the pair of reduced images can be designated as l′ and thedarker image of the pair of reduced images can be designated d′. Whilethis portion of process 500 may be referred to as “filtering” thedigital images to produce a reduced-resolution pair of images becausethe act of reducing the resolution of an image is sometimes referred toas low-pass filtering, there are many ways pixels of an image can be“grouped.” Basically any method that groups some number of adjacentpixels of the original by averaging or a similar technique into a valuefor a single pixel that is stored in memory can be referred to asfiltering to produce a reduced pair of images, even if electronicfiltering is not technically used and even if files representing eachreduced-resolution image in its entirety are not actually stored. Inaddition to being used so that fewer “pixels” need to be involved inexposure calculations, reduced images d′ and l′ are used for calculatinga representative color for the neighborhood around each pixel in d andl.

It has been found that a good reduction factor to use for thereduced-resolution digital images for example implementations is between5 and 25 times, depending on how large the original images are. As anexample, the techniques described herein have been applied where theoriginal digital images are 12 megapixel images from a mobile phonecamera and the reduced-resolution pair of images includes 2.2 megapixelimages, a reduction factor of about 5.5. Large photos from modern DSLRand digital medium format cameras can require a reduction factor of 20to 25 times. Higher resolution images of the future will require greaterreduction factors.

Still referring to FIG. 5, at block 510, at least one pairwise set ofscale factors is gathered from the reduced-resolution images. In oneexample, two pairwise sets are gathered. A first pairwise set S₁ definesscale factors for light tones and a second pairwise set S₀ defines scalefactors for dark tones. Thus, tonal values from both images are used, asshown below:S ₁ ={lum(l′ _(xy))/lum(d′ _(xy))|inrange(d′ _(xy) ,l′ _(xy)) AND lum(d′_(xy))>=τ}S ₀ ={lum(l′ _(xy))/lum(d′ _(xy))|inrange(d′ _(xy) ,l′ _(xy)) AND lum(d′_(xy))<τ},where i_(xy) is a pixel at location (x, y) in image i. The range of xand y is [0,1], where 0 maps to the first pixel and 1 maps to the lastpixel of each respective dimension in i. When resolving i_(xy) to anactual pixel, it can be assumed some filtering is applied, for examplebilinear filtering. τ is a camera sensor dependent parameter thatindicates the luminance threshold of the image sensor for dark tones.That is, the luminance after which at higher luminance the sensorresponse is at least approximately linear. In this example, theparameter τ is used to eliminate some sets of pixels from being used inthe remaining calculations. lum(p) is the linear measure of theluminance of pixel p and inrange(p,q) occurs where undersaturated(p) ANDundersaturated(q) AND lum(q)/lum(p)<λe AND lum(q)/lum(p)>e/λ. λ is atuning parameter in [1, ∞), which dictates to what extent sampled scalefactors can deviate from the estimate e. The use of this tuningparameter helps avoid bad samples due to ghosting between the images.undersaturated(p) is true if and only if all channels are undersaturatedfor pixel p. A tuning parameter of 0.001 has been found to produce goodexposure normalization. However, a good tuning parameter for aparticular imaging setup can be determined iteratively by beginning witha value such as this one and determining when good normalization isachieved. If the guess is too high, one finds that the shadows getscaled too much. If the guess is too low, one ends up with a light-toneempirical scale factor that is too low (assuming there are lots ofshadows). For a 16-bit image (0.65535), 0.001 demarcates tones on eitherside of 66.

Still on FIG. 5, a light-tone representative scale factor is calculatedfor the pairwise sample set for light tones at block 512. In oneexample, a dark-tone representative scale factor is calculated for thepairwise sample set for dark tones at block 514. The use of tworepresentative scale factors is optional as will be discussed in furtherdetail at the end of the discussion of FIG. 5 herein. In this example,the statistical median of the sample set for light tones is used for thelight-tone representative scale factor rp and the statistical median forof the sample set for dark tones is used for the dark-tonerepresentative scale factor no.

At block 516, process 500 selectively interpolates between thelight-tone representative scale factor η₁ and a comparative scale factorto produce an updated value for η₁, which can then be referred to as alight-tone empirical scale factor. In this example, the comparativescale factor is the normal scale factor e. By “selectively interpolate”what is meant is that interpolation is performed, or not, depending onthe number of good samples from the corresponding set of pairwisesamples. Since some sample pairs are not considered inrange, or areeliminated using the parameter τ, the number of “good” samples in thepair of reduced images varies. If the number of good samples is toosmall, the empirical scale factor is set to the normal scale factor e tocompensate for the small number of samples. If the value is sufficientlylarge, the empirical scale factor keeps the value of the representativescale factor η₁. Otherwise, the value is obtained from interpolationbetween the value of the representative scale factor and the normalscale factor e, as shown below:η₁ =eif |S ₁|/(w′h′)≤V ₀η₁=interp(e,η ₁,(|S ₁|/(w′h′)−V ₀)/(V ₁ −V ₀)) if V ₀ <|S ₁|/(w′h′)≤V ₁η₁=η₁ if V ₁ <|S ₁|/(w′h′),where the function interp(a,b,t) can be any standard interpolationfunction, which interpolates between a and b based on t∈[0,1]. V₀ and V₁are proportional sample sizes, which determine weights for therepresentative and empirical scale factors. V₁>V₀.

Still referring to FIG. 5, at block 514, a dark-tone empirical scalefactor can be calculated by selectively interpolating between scalefactors. In this case, the system selectively interpolates between thelight-tone representative scale factor, the updated value of η₁, and adark-tone representative scale factor η₀ at block 518 as shown below toupdate the value no to that of the dark-tone empirical scale factor asshown below, since once calculated, η₁ is the most reliable scale factoravailable.η₀=η₁if |S ₀|/(w′h′)≤V ₀η₀=interp(η₁,η₀,(|S ₀|/(w′h′)−V ₀)/(V ₁ −V ₀)) if V ₀ <|S ₀|/(w′h′)≤V ₁η₀=η₀ if V ₁ <|S ₀|/(w′h′),

In one example, prior to applying the empirical scale factors, a versionof the dark image d is produced that is smaller in resolution than d. Inthis example, d is downsized to d″ of size w″−h″. d″ may be larger orsmaller in resolution than d′, or could be the same resolution, but willalways be lower resolution than d. This version of d serves the functionof allowing a representative color for a neighborhood around pixel p tobe calculated for use in applying the function described below.

The empirical scale factors are then applied to dark image d at block522 to yield an empirically normalized version of d for use in furtherprocessing, at block 524. Further processing might include repeatingwith additional images, combining images via HDR processing, applying ade-ghosting process, or any other process that can benefit from use ofan empirically normalized digital image. In this example, the empiricalscale factors are applied by defining and applying a function fpixelwise to d as shown below.f(d _(xy))=η₀ d _(xy) if lum(d″ _(xy))≤t _(min)f(d _(xy))=interp(η₀ d _(xy),η₁ d _(xy),(lum(d″ _(xy))−t _(min))/(τ−t_(min))) if t _(min) <lum(d″ _(xy))≤τf(d _(xy))=η₁ d _(xy) if τ<lum(d″ _(xy)),where t_(min)=min{lum(d′_(xy))}.

FIG. 6 shows a graph 600 that illustrates the shape of function f Line602 in graph 600 is the shape of function f based on the equationsimmediately above using both empirical scale factors. Line 604immediately above line 602 is the shape the function would take on ifbased only the light-tone empirical scale factor. Line 606, immediatelybelow line 602 is the shape the function would take if based solely onthe dark-tone empirical scale factor. Line 608, the highest line on thegraph, shows the luminance if based on a normal scale factor determinedfrom EXIF data. Note that after linearization, recomputing the responsebetween the linearized dark image and the light image should now producea mapping that if plotted would look closer 604 (where it was 602 beforethe linearization)—i.e. the kink at the bottom of the line is removed.

It is possible to empirically normalize an image by using only oneempirical scale factor and interpolating between one representativescale factor and the normal scale factor to determine the one empiricalscale factor, which would be the equivalent of the light-tone empiricalscale factor discussed above. In this case, the camera sensor luminancethreshold is not taken into account. Process 500 of FIG. 5 would notinclude the blocks in dashed lines. These results would deviate from theresult of applying the function immediately above, especially at lowerluminance values as shown by line 604 of FIG. 6. However, theempirically normalized digital image would still be an improvement overrelying on a normal scale factor based solely on EXIF data. In thiscase, mathematically, since no pixel can have luminance less than zero,the set of samples for the dark tones would be empty meaning that thedark tone scale factor η₀ would be set to η₁ (during theinterpolation/correction step) and hence all pixels will be scaleduniformly by η₁. It should be noted that some image formats, for examplethe DNG format, do support negative values, that is, values, which areless than the so-called “Blacklevel.” For purposes of this disclosure ithas been assumed that the images are already at least demosaiced andlinearized, with pixel values greater than or equal to zero.

The process described above works with digital image pairs, and in manycases that is all that is needed since it will only be the darkest imageof a group of images that needs to be corrected, in which case the twolowest ev images of the group can be used. However, the process can beapplied to larger numbers of digital images by applying it pairwise morethan once. The process is performed once to produce an empiricallynormalized digital image and then repeated wherein the pair of digitalimages used in the process as repeated includes the empiricallynormalized digital image. In such a case, the process can be applied tothe higher ev of the “underexposed” images using that image and the nextbrightest above it to produce a normalized image for the higher ev ofthe underexposed images, then the previously normalized digital imagewould be used as the “light” image with the next lower ev of the“underexposed” images and so forth.

Numerous specific details are set forth herein to provide a thoroughunderstanding of the claimed subject matter. However, those skilled inthe art will understand that the claimed subject matter may be practicedwithout these specific details. In other instances, methods,apparatuses, or systems that would be known by one of ordinary skillhave not been described in detail so as not to obscure claimed subjectmatter.

Unless specifically stated otherwise, it is appreciated that throughoutthis specification discussions utilizing terms such as “processing,”“computing,” and “calculating” or the like refer to actions or processesof a computing device, such as one or more computers or a similarelectronic computing device or devices, that manipulate or transformdata represented as physical electronic or magnetic quantities withinmemories, registers, or other information storage devices, transmissiondevices, or display devices of the computing platform.

The system or systems discussed herein are not limited to any particularhardware architecture or configuration. A computing device can includeany suitable arrangement of components that provides a resultconditioned on one or more inputs. Suitable computing devices includemultipurpose microprocessor-based computer systems accessing storedsoftware that programs or configures the computing system from a generalpurpose computing apparatus to a specialized computing apparatusimplementing one or more embodiments of the present subject matter. Anysuitable programming, scripting, or other type of language orcombinations of languages may be used to implement the teachingscontained herein in software to be used in programming or configuring acomputing device.

Embodiments of the methods disclosed herein may be performed in theoperation of such computing devices. The order of the blocks presentedin the examples above can be varied—for example, blocks can bere-ordered, combined, and/or broken into sub-blocks. Certain blocks orprocesses can be performed in parallel.

The use of “configured to” herein is meant as open and inclusivelanguage that does not foreclose devices adapted to or configured toperform additional tasks or steps. Additionally, the use of “based on”is meant to be open and inclusive, in that a process, step, calculation,or other action “based on” one or more recited conditions or values may,in practice, be based on additional conditions or values beyond thoserecited. Headings, lists, and numbering included herein are for ease ofexplanation only and are not meant to be limiting.

While the present subject matter has been described in detail withrespect to specific embodiments thereof, it will be appreciated thatthose skilled in the art, upon attaining an understanding of theforegoing, may readily produce alterations to, variations of, andequivalents to such embodiments. Accordingly, it should be understoodthat the present disclosure has been presented for purposes of examplerather than limitation, and does not preclude inclusion of suchmodifications, variations, and/or additions to the present subjectmatter as would be readily apparent to one of ordinary skill in the art.

What is claimed is:
 1. A method comprising: acquiring, by a processingdevice, a dark exposure of a first digital image and a light exposure ofa second digital image, wherein the first digital image includes a firstphotographic content item and the second digital image includes a secondphotographic content item; determining, by the processing device, thatthe first photographic content item matches the second photographiccontent item; producing, by the processing device, a firstreduced-resolution image from the first digital image and a secondreduced-resolution image from the second digital image; calculating, bythe processing device, a light-tone representative scale factor and adark-tone representative scale factor for each of the firstreduced-resolution image and the second reduced-resolution image;producing, by the processing device, a dark-tone empirical scale factorby selectively interpolating between the dark-tone representative scalefactor and the light-tone representative scale factor for each of thefirst reduced-resolution image and the second reduced-resolution image;and generating, by the processing device, an empirically normalizeddigital image by applying, pixelwise, a function including at least thedark-tone empirical scale factor to the first digital image or thesecond digital image.
 2. The method of claim 1, wherein the calculatingof the dark-tone representative scale factor for each of the firstreduced-resolution image and the second reduced-resolution image furthercomprises gathering a dark-tone pairwise set of scale factors from thefirst reduced-resolution image and the second reduced-resolution image.3. The method of claim 1, wherein selectively interpolating comprisesinterpolating among the light-tone representative scale factor, anupdated value of the light-tone representative scale factor, and thedark-tone representative scale factor to update a value of the dark-toneempirical scale factor.
 4. The method of claim 3, further comprisinginterpolating a normal scale factor based at least in part onexchangeable image file data.
 5. The method of claim 1, wherein thefunction includes a camera sensor parameter indicating a luminancethreshold of a camera sensor for dark tones.
 6. The method of claim 5,wherein the luminance threshold is a luminance value above which acamera sensor response is at least approximately linear.
 7. A systemcomprising: a processing device; and a non-transitory computer-readablemedium coupled to the processing device, wherein the processing deviceis configured to execute computer program code stored in thenon-transitory computer-readable medium and thereby perform operationscomprising: producing a first reduced-resolution image from a firstdigital image and a second reduced-resolution image from a seconddigital image, wherein a first exposure value of the first digital imageis greater than a second exposure value of the second digital image, andwherein the first digital image includes a first photographic contentitem that matches a second photographic content item included in thesecond digital image; calculating a light-tone representative scalefactor and a dark-tone representative scale factor for each of the firstreduced-resolution image and the second reduced-resolution image;producing a dark-tone empirical scale factor by selectivelyinterpolating between the dark-tone representative scale factor and thelight-tone representative scale factor for each of the firstreduced-resolution image and the second reduced-resolution image; andgenerating an empirically normalized digital image by applying afunction including the dark-tone empirical scale factor to each pixel ofthe first digital image or the second digital image.
 8. The system ofclaim 7, the operations further comprising calculating the dark-tonerepresentative scale factor by using a dark-tone pairwise set of scalefactors from the first reduced-resolution image and the secondreduced-resolution image.
 9. The system of claim 7, wherein selectivelyinterpolating comprises interpolating among the light-tonerepresentative scale factor, an updated value of the light-tonerepresentative scale factor, and the dark-tone representative scalefactor to update a value of the dark-tone empirical scale factor. 10.The system of claim 9, the operations further comprising interpolating anormal scale factor based at least in part on exchangeable image filedata.
 11. The system of claim 7, wherein the function includes a camerasensor parameter that indicates a luminance threshold of a camera sensorfor dark tones.
 12. The system of claim 11, wherein the luminancethreshold is a luminance value above which a camera sensor response isat least approximately linear.
 13. The system of claim 7, wherein thefirst reduced-resolution image from the first digital image and thesecond reduced-resolution image from the second digital image each havea resolution reduction factor between 5 and
 25. 14. A non-transitorycomputer-readable medium storing program code executable by a processingdevice to perform operations, the operations comprising: acquiring adark exposure digital image and a light exposure digital image, whereinthe dark exposure digital image includes a photographic content itemthat is also included in the light exposure digital image; producing afirst reduced-resolution image from the dark exposure digital image anda second reduced-resolution image from the light exposure digital image;calculating a light-tone representative scale factor and a dark-tonerepresentative scale factor for each of the first reduced-resolutionimage and the second reduced-resolution image; producing a dark-toneempirical scale factor by selectively interpolating between thedark-tone representative scale factor and the light-tone representativescale factor for each of the first reduced-resolution image and thesecond reduced-resolution image; and generating an empiricallynormalized digital image by applying, pixelwise, a function includingthe dark-tone empirical scale factor to the dark exposure digital imageand the light exposure digital image.
 15. The non-transitorycomputer-readable medium of claim 14, the operations further comprisingcalculating the dark-tone representative scale factor by using adark-tone pairwise set of scale factors from the firstreduced-resolution image and the second reduced-resolution image. 16.The non-transitory computer-readable medium of claim 14, wherein thefunction includes a camera sensor parameter that indicates a luminancethreshold of a camera sensor for dark tones.
 17. The non-transitorycomputer-readable medium of claim 16, wherein the luminance threshold isa luminance value above which a camera sensor response is at leastapproximately linear.
 18. The non-transitory computer-readable medium ofclaim 14, wherein selectively interpolating comprises an interpolationof the light-tone representative scale factor, an updated value of thelight-tone representative scale factor, and the dark-tone representativescale factor to update a value of the dark-tone empirical scale factor.19. The non-transitory computer-readable medium of claim 18, theoperations further comprising interpolating a normal scale factor basedat least in part on exchangeable image file data.
 20. The non-transitorycomputer-readable medium of claim 14, wherein the firstreduced-resolution image from the dark exposure digital image and thesecond reduced-resolution image from the light exposure digital imageeach have a resolution reduction factor between 5 and 25.